Synchronization of logical copy relationships

ABSTRACT

An approach to synchronization of logical copy relationships on a local site with those on a remote site that is a mirror. A withdraw command is received on the local site for logical copy relationships between local source tracks and local target tracks. The withdraw command is executed and sent to a remote site buffer, from which it will be transferred to the remote site. Change indicators are set for the at least one local track in the affected logical copy relationships. If the withdraw command is lost from the remote site buffer, the remote site buffer will be rebuilt using the change indicators. By setting change indicators for local tracks in a withdrawn logical copy relationship (even if the data in local tracks hasn&#39;t changed), the remote counterparts to these tracks on the remote site will be rewritten, resulting in withdrawal of the logical copy relationship there even if the withdrawal command was never sent to the remote site.

FIELD

The subject matter disclosed herein relates to maintainingsynchronization of logical copy relationships on a local site withlogical copy relationships on a remote site.

BACKGROUND Description of the Related Art

The use of “mirrors”, or copies of data, plays an important role in manycomputing environments. Where data integrity and recovery is important,using a computing system to provide a mirror of another computing systemcan be a valuable tool. In many situations, a local site has anassociated remote site that acts as a mirror.

Some mirroring environments use asynchronous mirroring; that is, whenchanges are made on the local site, they are not simultaneously executedon the mirror site. Rather, data representing the changes may bebuffered at the local site and then transmitted for implementation onthe remote site. In such environments, the data in the buffers may belost when the mirroring session between the local site and remote siteis suspended, whether the suspension is intention or unintentional. Thelocal site may maintain data structures to track which tracks of datahave changed over a certain period of time, and use those datastructures to rebuild the buffer after a suspend. In certain systems,rather than rebuild the buffer, the system may use a logical indicatorfor what data needs to be copied to the mirror site.

Certain environments can also take advantage of logical copy operations,such as IBM's FlashCopy. A logical copy operation is a process thatcreates a logical copy of data by associating data stored at a sourcelocation (which is the data being copied) with a target location (whichis the copy of the data). The logical copy operation may, for example,use pointers to establish a relationship between the data at the sourcelocation and the copy of that data at the target location withoutactually copying data from the source to the target. When the copy isdeleted, the logical copy relationships may be deleted to effectivelydelete the copy.

Difficulties may arise where a logical copy operation is executed on thelocal site (establishing logical copy relationships on the local site)and then mirrored to the remote site (establishing corresponding logicalcopy relationships on the remote site). If the user withdraws thelogical copy relationships on the local site (for example, by deletingthe copy), and that withdraw command is lost due to a suspension whilethe withdraw command was in the buffer, then the local site and remotesite may be out of synch since the withdraw command is never propagatedto the remote site.

BRIEF SUMMARY

The present invention allows for synchronization of logical copyrelationships between a local site and a remote site. The invention maybe realized as an apparatus, a computer program product, a method, asystem, or in other forms.

A computer program product stored on a computer readable medium may beconfigured to receive a withdraw command initiated by a user for alogical copy relationship between a local source track and a localtarget track on the local site. The withdraw command may cause thelogical copy relationship between the local source track and the localtarget track to terminate on the local site. The withdraw command may befor a plurality of logical copy relationships between local sourcetracks and local target tracks. The computer program product may also beconfigured to send the withdraw command to a remote site buffer. Thewithdraw command may be read from the remote site buffer and sent to theremote site for execution if there is no suspension which causes thewithdraw command to be lost from the remote site buffer.

The computer program product may also be configured to set a changeindicator that indicates a change in data for one or both of the localsource track and the local target track in the logical copy relationshipthat is the subject of the withdraw command. The change indicator may bea bit entry in a bitmap that contains one bit entry for each local trackin a volume that the bitmap represents. The change indicator istypically used to indicate a change in data. The computer programproduct may take this action in response to receiving the withdrawcommand for the logical copy relationship.

The computer program product may be configured to detect a suspension ofthe mirror relationship between the local site and the remote site thatcauses the withdraw command in the remote site buffer to be discarded.It may also determine that data in those local tracks that have thechange indicator set may be inconsistent at the remote site. Localtracks are tracks on the local site, and may be local source tracksand/or local target tracks.

The computer program product may send data in the local tracks that havethe change indicator set to the remote site to be written atcorresponding remote tracks. As a result, remote logical copyrelationships between remote source tracks and remote target tracks willbe withdrawn at the remote site.

In certain embodiments, the computer program product may determine ifthe establish command that created the logical copy relationshipincluded a background copy command. If so, it may refrain from settingthe change indicator when it receives the withdrawal command.

The invention may also be a method that includes the steps of receivingthe withdraw command initiated by the user for the logical copyrelationship between the local source track and the local target trackon the local site, and sending the withdraw command to the remote sitebuffer. The method may also involve setting a change indicator for oneor both of the local source track and the local target track in theaffected logical copy relationship. The method may involve detecting asuspension of the mirror relationship that causes the withdraw commandin the remote site buffer to be discarded, determining which localtracks have the change indicators set, and sending data in those localtracks to be written at the corresponding remote tracks at the remotesite. The result is that data in at least one of the local source trackand local target track is written at the remote site, which terminatesthe remote logical copy relationship at the remote site.

The invention may also be a system that includes a mirror module thatcreates a mirror relationship between the local site and the remotesite, and that stores change data representing changes to local data inthe remote site buffer on the local site. The change data may be thechanged data itself, metadata, or any data sufficient to allow theremote site to properly mirror the change on the remote site.

The system may also use a bitmap for storage devices on the local site.The storage devices may include local tracks, and each local track maybe associated with a bit entry in the bitmap. The bit entries may be thechange indicators. A receipt module may receive the withdraw commandinitiated by the user for a set of logical copy relationships betweenlocal source tracks and local target tracks on the storage devices. Themirror module may send the withdraw command to the remote site buffer,and may be configured to transfer that withdraw command to the remotesite. A backup module may set bit entries in the bitmap for the localtarget tracks in the logical copy relationships subject to the withdrawcommand. The mirror module may be configured to detect suspensionscausing the withdraw command to be discarded from the remote sitebuffer, and, in response, to retrieve data from the local target trackswith the bit entries set in the bitmap, and to then send the data in thelocal target tracks to be written at the remote site.

The invention may also be realized as an apparatus that includes areceipt module that receives the withdraw command initiated by the userfor a plurality of logical copy relationships between local sourcetracks and local target tracks. A transfer module may send the withdrawcommand to the remote site buffer. A backup module may set changeindicators for the local target tracks in each logical copy relationshipsubject to the withdraw command. The change indicator may be a bit entryin a bitmap for maintaining synchronization between the local site andthe remote site. Setting the change indicator may mean flipping the bitfrom one value to another; for example, from 0 to 1, or 1 to 0.

These features and advantages of the embodiments will become more fullyapparent from the following description and appended claims, or may belearned by the practice of embodiments as set forth hereinafter.

BRIEF DESCRIPTION OF THE DRAWINGS

In order that the advantages of the embodiments of the invention will bereadily understood, a more particular description of the embodimentsbriefly described above will be rendered by reference to specificembodiments that are illustrated in the appended drawings. Understandingthat these drawings depict only some embodiments and are not thereforeto be considered to be limiting of scope, the embodiments will bedescribed and explained with additional specificity and detail throughthe use of the accompanying drawings, in which:

FIG. 1 is a schematic block diagram illustrating one embodiment of asystem in a mirroring environment;

FIG. 2 is a schematic block diagram illustrating an embodiment of changeindicators in a mirroring environment;

FIGS. 3A and 3B are schematic block diagrams illustrating an embodimentof a logical copy operation establishing logical copy relationships;

FIG. 4 is a schematic block diagram illustrating an embodiment of alogical copy synchronization apparatus;

FIG. 5 is a schematic block diagram illustrating an embodiment of a casewhere a withdraw command is received;

FIGS. 6A, 6B, and 6C are schematic block diagrams illustrating anembodiment of a case where a withdraw command is received, a suspensionoccurs, and the remote logical copy operation is still withdrawn; and

FIG. 7 is a flow chart diagram illustrating a method for synchronizationof logical copy relationships.

DETAILED DESCRIPTION

As will be appreciated by one skilled in the art, aspects of the presentinvention may be embodied as a system, method or computer programproduct. Accordingly, aspects of the present invention may take the formof an entirely hardware embodiment, an entirely software embodiment(including firmware, resident software, micro-code, etc.) or anembodiment combining software and hardware aspects that may allgenerally be referred to herein as a “circuit,” “module” or “system.”Furthermore, aspects of the present invention may take the form of acomputer program product embodied in one or more computer readablemedium(s) having computer readable program code embodied thereon.

Many of the functional units described in this specification have beenlabeled as modules, in order to more particularly emphasize theirimplementation independence. For example, a module may be implemented asa hardware circuit comprising custom VLSI circuits or gate arrays,off-the-shelf semiconductors such as logic chips, transistors, or otherdiscrete components. A module may also be implemented in microcode,firmware, or the like of programmable hardware devices such as fieldprogrammable gate arrays, programmable array logic, programmable logicdevices or the like.

Modules may also be implemented in software for execution by varioustypes of processors. An identified module of computer readable programcode may, for instance, comprise one or more physical or logical blocksof computer instructions which may, for instance, be organized as anobject, procedure, or function. Nevertheless, the executables of anidentified module need not be physically located together, but maycomprise disparate instructions stored in different locations which,when joined logically together, comprise the module and achieve thestated purpose for the module.

Indeed, a module of computer readable program code may be a singleinstruction, or many instructions, and may even be distributed overseveral different code segments, among different programs, and acrossseveral memory devices. Similarly, operational data may be identifiedand illustrated herein within modules, and may be embodied in anysuitable form and organized within any suitable type of data structure.The operational data may be collected as a single data set, or may bedistributed over different locations including over different storagedevices, and may exist, at least partially, merely as electronic signalson a system or network. Where a module or portions of a module areimplemented in software, the computer readable program code may bestored and/or propagated on in one or more computer readable medium(s).

The computer readable medium may be a tangible computer readable storagemedium storing the computer readable program code. The computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, holographic,micromechanical, or semiconductor system, apparatus, or device, or anysuitable combination of the foregoing.

More specific examples of the computer readable medium may include butare not limited to a portable computer diskette, a hard disk, a randomaccess memory (RAM), a read-only memory (ROM), an erasable programmableread-only memory (EPROM or Flash memory), a portable compact discread-only memory (CD-ROM), a digital versatile disc (DVD), a Blu-RayDisc (BD), an optical storage device, a magnetic storage device, aholographic storage medium, a micromechanical storage device, or anysuitable combination of the foregoing. In the context of this document,a computer readable storage medium may be any tangible medium that cancontain, and/or store computer readable program code for use by and/orin connection with an instruction execution system, apparatus, ordevice.

The computer readable medium may also be a computer readable signalmedium. A computer readable signal medium may include a propagated datasignal with computer readable program code embodied therein, forexample, in baseband or as part of a carrier wave. Such a propagatedsignal may take any of a variety of forms, including, but not limitedto, electrical, electro-magnetic, magnetic, optical, or any suitablecombination thereof. A computer readable signal medium may be anycomputer readable medium that is not a computer readable storage mediumand that can communicate, propagate, or transport computer readableprogram code for use by or in connection with an instruction executionsystem, apparatus, or device. Computer readable program code embodied ona computer readable signal medium may be transmitted using anyappropriate medium, including but not limited to wireless, wireline,optical fibre cable, Radio Frequency (RF), or the like, or any suitablecombination of the foregoing.

In one embodiment, the computer readable medium may comprise acombination of one or more computer readable storage mediums and one ormore computer readable signal mediums. For example, computer readableprogram code may be both propagated as an electro-magnetic signalthrough a fibre optic cable for execution by a processor and stored onRAM storage device for execution by the processor.

Computer readable program code for carrying out operations for aspectsof the present invention may be written in any combination of one ormore programming languages, including an object oriented programminglanguage such as Java, Smalltalk, C++ or the like and conventionalprocedural programming languages, such as the “C” programming languageor similar programming languages. The computer readable program code mayexecute entirely on the user's computer, partly on the user's computer,as a stand-alone software package, partly on the user's computer andpartly on a remote computer or entirely on the remote computer orserver. In the latter scenario, the remote computer may be connected tothe user's computer through any type of network, including a local areanetwork (LAN) or a wide area network (WAN), or the connection may bemade to an external computer (for example, through the Internet using anInternet Service Provider).

Reference throughout this specification to “one embodiment,” “anembodiment,” or similar language means that a particular feature,structure, or characteristic described in connection with the embodimentis included in at least one embodiment. Thus, appearances of the phrases“in one embodiment,” “in an embodiment,” and similar language throughoutthis specification may, but do not necessarily, all refer to the sameembodiment, but mean “one or more but not all embodiments” unlessexpressly specified otherwise. The terms “including,” “comprising,”“having,” and variations thereof mean “including but not limited to,”unless expressly specified otherwise. An enumerated listing of itemsdoes not imply that any or all of the items are mutually exclusive,unless expressly specified otherwise. The terms “a,” “an,” and “the”also refer to “one or more” unless expressly specified otherwise.

Furthermore, the described features, structures, or characteristics ofthe embodiments may be combined in any suitable manner. In the followingdescription, numerous specific details are provided, such as examples ofprogramming, software modules, user selections, network transactions,database queries, database structures, hardware modules, hardwarecircuits, hardware chips, etc., to provide a thorough understanding ofembodiments. One skilled in the relevant art will recognize, however,that embodiments may be practiced without one or more of the specificdetails, or with other methods, components, materials, and so forth. Inother instances, well-known structures, materials, or operations are notshown or described in detail to avoid obscuring aspects of anembodiment.

Aspects of the embodiments are described below with reference toschematic flowchart diagrams and/or schematic block diagrams of methods,apparatuses, systems, and computer program products according toembodiments of the invention. It will be understood that each block ofthe schematic flowchart diagrams and/or schematic block diagrams, andcombinations of blocks in the schematic flowchart diagrams and/orschematic block diagrams, can be implemented by computer readableprogram code. These computer readable program code may be provided to aprocessor of a general purpose computer, special purpose computer,sequencer, or other programmable data processing apparatus to produce amachine, such that the instructions, which execute via the processor ofthe computer or other programmable data processing apparatus, createmeans for implementing the functions/acts specified in the schematicflowchart diagrams and/or schematic block diagrams block or blocks.

The computer readable program code may also be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the schematic flowchart diagramsand/or schematic block diagrams block or blocks.

The computer readable program code may also be loaded onto a computer,other programmable data processing apparatus, or other devices to causea series of operational steps to be performed on the computer, otherprogrammable apparatus or other devices to produce a computerimplemented process such that the program code which execute on thecomputer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

The schematic flowchart diagrams and/or schematic block diagrams in theFigures illustrate the architecture, functionality, and operation ofpossible implementations of apparatuses, systems, methods and computerprogram products according to various embodiments of the presentinvention. In this regard, each block in the schematic flowchartdiagrams and/or schematic block diagrams may represent a module,segment, or portion of code, which comprises one or more executableinstructions of the program code for implementing the specified logicalfunction(s).

It should also be noted that, in some alternative implementations, thefunctions noted in the block may occur out of the order noted in theFigures. For example, two blocks shown in succession may, in fact, beexecuted substantially concurrently, or the blocks may sometimes beexecuted in the reverse order, depending upon the functionalityinvolved. Other steps and methods may be conceived that are equivalentin function, logic, or effect to one or more blocks, or portionsthereof, of the illustrated Figures.

Although various arrow types and line types may be employed in theflowchart and/or block diagrams, they are understood not to limit thescope of the corresponding embodiments. Indeed, some arrows or otherconnectors may be used to indicate only the logical flow of the depictedembodiment. For instance, an arrow may indicate a waiting or monitoringperiod of unspecified duration between enumerated steps of the depictedembodiment. It will also be noted that each block of the block diagramsand/or flowchart diagrams, and combinations of blocks in the blockdiagrams and/or flowchart diagrams, can be implemented by specialpurpose hardware-based systems that perform the specified functions oracts, or combinations of special purpose hardware and computer readableprogram code.

FIG. 1 shows one embodiment of a system 100 comprising a local site 102and a remote site 104 connected by a communications connection 160. Alocal site 102 is a group of one or more computers which store data on alocal storage device 120. The local storage device 120 may include oneor more physical devices, and may be any variety of storage device thatcan store data. For example, the local storage device 120 may includesolid state storage, hard disk drives, tape drives, and other varietiesof storage devices. Examples of local sites 102 include: a single userwith a single computer; a network of computers in a relatedorganization, such as a business or department area within a business;and a university or department within a university.

The remote site 104 is a group of one or more computers which can storedata on remote storage device 150. The remote storage device 150 mayalso be one or more physical devices, and may be any type of the manypossible storage device types. The remote site 104 may be configured toact as a mirror for the local site 102. A mirror, as used in thisapplication, is a system that stores a copy of the data stored at alocal site 102. The mirror may be, for example, a backup of the datastored at a local site 102. In certain embodiments, the remote site 104is located at a facility that is physically separate from the local site102; however, the remote site 104 need not be physically separate ordistant from the local site 102.

The local site 102 and the remote site 104 are connected by acommunications connection 160. A communications connection 160 is aconnection between the local site 102 and the remote site 104 whichallows the local site 102 and the remote site 104 to exchange data. Inone embodiment, the communications connection 160 is an Internetconnection. The communications connection 160 may also be also be anEthernet connection, a Fibre Channel connection, an Infinibandconnection, or other suitable connection.

The local site 102 may also include a mirror module 110, and a remotesite buffer 140. The mirror module 110 is configured to provideasynchronous data replication between the local site 102 and the remotesite 104. The mirror module 110 may include hardware and software tosupport asynchronous data replication between the local site 102 and theremote site 104. In certain embodiments, the mirror module 110 is IBM'sGlobal Mirror for z/Series (also commonly known as Extended Remote Copy,or XRC). In other embodiments, the mirror module 110 is IBM's Peer toPeer Remote Copy (PPRC) service.

While FIG. 1 shows the mirror module 110 implemented on the local site102, the mirror module 110 may be implemented on the remote site 104, oron both the local site 102 and the remote site 104. In certainembodiments, the mirror module 110 may include a system data mover thatresides on a host at the remote site 104. Thus, the mirror module 110may send data from the local site 102 to the remote site 104, or themirror module 110 on the remote site 104 may read data from the localsite 102. The mirror module 110 may use either a “push” or a “pull”approach (or some combination thereof) to moving data, and the presentinvention is not limited to any particular approach.

In certain embodiments, the local site 102 includes a remote site buffer140 that stores data and commands representing changes to the data onthe local site 102 which needs to be sent to the remote site 104. Forexample, when data is written to one or more tracks on the local storagedevice 120, that data may also be placed into the remote site buffer140. The mirror module 110 may then move that data from the remote sitebuffer 140 to the remote site 104, where the data may be stored inappropriate tracks.

The remote site buffer 140 may be a first in first out (FIFO) buffer onthe local site 102. In certain embodiments, the remote site buffer 140may be known as a sidefile and be part of a controller on the local site102. In certain embodiments, at least a portion of the remote sitebuffer 140 is implemented in volatile memory. In such embodiments, datain the remote site buffer 140 may be lost if there is a suspension inthe mirroring operations being performed. Thus, in certain embodiments,data may be lost from the remote site buffer 140 in the event of ahardware failure, an interrupt, a problem in the communicationsconnection 160, an intentional suspension of the mirroring operations,or other even which causes loss of data in the remote site buffer 140.

The local site 102 may also contain a logical copy module 160 that cancreate and maintain logical copies of data sets. The logical copy module160 may support a logical copy operation 318 (discussed below) that is aprocess that creates a logical copy of data by associating data storedat a source location (which is the subject of the logical copy operation318) with a target location. Thus, a user may request to copy a dataset, which copy is named data set B. The logical copy module 160 mayrespond by indicating that it has created a copy called data set B. Infact, rather than physically copy the data set A to a new location, thelogical copy module 160 may have created a logical copy of data byassociating the location for data set B with the data in data set A. Inone embodiment, the logical copy module 160 is IBM's FlashCopy module.Logical copy is discussed in greater detail below in connection withFIG. 3.

FIG. 2 shows one embodiment of a system 200 with the local storagedevice 120 and a change detection module 210. The local storage device120, as explained above, may be any medium suitable for storing data onthe local site 102. FIG. 2 depicts the local storage device 120 asincluding local tracks 202 a-h. A local track 202 is a unit of datastored on the local storage device 120. The term “track” is used broadlyin this application to encompass various possible units of data, and isnot used in a way which limits the term to storage on hard disk drives,or to a particular hardware configuration of a hard disk drive. Thus,the local tracks 202 may be physical tracks on a hard disk drive, unitsof storage on a tape drive, or other units for storage on local storagedevice 120 which may vary with the media used to implement the localstorage device 120. Similarly, there may be multiple local tracks 202for each physical track of a hard disk drive on the local storage device120. The local track may be a source track 312 or a target track 314,both of which are discussed in more detail in connection with FIG. 3 b.

The change detection module 210 sets a change indicator 204 for acorresponding local track 202. The change indicator 204 indicateswhether the local track 202 has been changed. The change indicator 204may indicate that the associated local track 202 may be out ofsynchronization with the remote site 104.

For example, the data in local track 202 a may be updated with a newvalue. In response to this change, the change detection module 210 mayset the change indicator 204 a, and the mirror module 110 may send thedata to the remote site buffer 140 to be written to the remote site 104.In the event of a suspension, the remote site buffer 140 may lose thedata representing the change to the local track 202 a before the datacan be sent to the remote site 104. In certain embodiments, the mirrormodule 110 may use the change indicators 204 to determine which localtracks 202 may have been changed, but have not had those changes sent tothe remote site 104.

In one embodiment, the change detection module 210 may build andmaintain one or more bitmaps for each volume in the local storage device120 to store the change indicators 204. The change indicators 204 may bebits in the bitmap. When the local track 202 a is changed, the changedetection module 210 set the change indicator 204 a by flipping a bitassociated with the local track 202 a in the bitmap. In this way, thechange detection module 210 may monitor which tracks contain data thatis out of synch between the local site 102 and the remote site 104. Inthe event that there is an event which causes the data in the remotesite buffer 140 go be lost, the change indicators 204 may be used toensure that changes made to data in the local tracks 202 are reflectedon the remote site 104.

In certain embodiments, the change detection module 210 maintains morethan one bitmap. For example, the change detection module 210 maymaintain two bitmaps for each volume in the local storage device 120.The change detection module 210 may maintain a current bitmap (which maybe referred to as bitmap n) and a previous bitmap (which may be referredto as bitmap n−1). As changes are made to data in the local tracks 202,the changes are reflected in the associated change indicators 204. Aftera period of time, the change detection module 210 may change thedesignation of the current bitmap to previous bitmap. The changedetection module 210 may then reset the old version of the previousbitmap and use this as the new current bitmap.

For example, the change detection module 210 may use the current bitmapfor five minutes. Once the five minutes are complete, the current bitmapmay be designated the previous bitmap, and the previous bitmap may bereset and become the new current bitmap. Once an additional five minutespass, the bitmaps are once again toggled. In such an embodiment, thechange detection module 210 may maintain change indicators 204 for localtracks 202 for a five to ten minute window. In such embodiments, in theevent of a failure, the mirror module 110 may send the data in all localtracks 202 having the change indicator 204 to be written at the remotesite 104. Embodiments other than the one described above may also beimplemented.

FIGS. 3 a and 3 b illustrate one embodiment of the operation of alogical copy operation 318. A logical copy operation 318 is a processthat creates a logical copy of data by associating data stored at asource location (which is the subject of the logical copy operation)with a target location. The logical copy operation 318 creates a logicalcopy of a source data set 310 in a target data set 320. A data set, asused in this application, is any logical or physical grouping of data.For example, a data set may be a volume, a data set, a range of tracks,or other data grouping. FIG. 3A shows an example where a source data set310 located at a source location is the subject of a logical copyoperation 318 that creates a logical copy of the data set (called thetarget data set 320) at a target location. The logical copy operationmay be used to create a point in time copy of one or more logicalvolumes and/or data sets. The target data set 320 may be used as abackup copy, a copy of a source data set 310 to be used for developmentwithout affecting the integrity of the source data set 310, or for otherreasons. One example of a logical copy operation is IBM's FlashCopyoperation.

FIG. 3 b illustrates one embodiment of the result of executing a logicalcopy operation 318 against a source data set 310 causing the creation ofa logical copy of the data at the target data set 320. In oneembodiment, the logical copy operation causes one or more logical copyrelationships 316 a-d to be created, which logical copy relationshipsassociate the source tracks 312 a-d of the source data set 310 with thetarget tracks 314 a-d of the target data set 320.

In one embodiment, the logical copy relationships 316 a-d are pointers.In such an embodiment, the logical copy operation 318 may createpointers between the source tracks 312 and the target tracks 314. Thus,when an application requests data from the target track 314 a, therequest may be redirected by the pointer to the source track 312 a.Other approaches to creating a logical copy relationship 316 may also beused. Creating logical copy relationships 316 which associate the sourcetracks 312 with the target tracks (instead of physically moving the datafrom source tracks 312 to target tracks 314) may allow a user and/orapplication to copy a source data set 310 far more quickly than wouldoccur if the data in the source tracks 312 had to be physically moved tothe target tracks 314 in order to make the copy.

In certain embodiments, the logical copy relationships 316 are adjustedto account for changes to the source tracks 312 and/or the target tracks314. For example, after the target data set 320 (which is the logicalcopy of source data set 310) is made, an application may write data totarget track 314 c of the target data set 320. In response, the data maybe written to target track 314 c and the logical copy relationship 316 cwithdrawn. Withdrawing a logical copy relationship 316 means that thelogical copy relationship between the source track 312 and the targettrack 314 is terminated. For example, a logical copy relationship 316may be withdrawn by deleting or otherwise removing a pointer thatassociates source track 312 c and target track 314 c. To continue theexample given above, future requests for the data in the target track314 c will use the data in target track 314 c, and will not be forwardedto source track 312 c.

In another example, after the source data set 310 is copied to targetdata set 320, a user or application may write new data to the sourcetrack 312 d. Before the data in source track 312 d is changed, the datamay be physically copied from source track 312 to target track 314 d.The data may then be written to the source track 312 d. As the data insource track 312 d has been physically moved to target track 314 d, thelogical copy relationship 316 d may also be withdrawn.

In certain embodiments, the logical copy module 160 issues a withdrawcommand for the affected logical copy relationship 316. Such withdrawcommands may be hidden from the user. For example, where the logicalcopy relationships 316 are withdrawn due to changes to either a sourcetrack 312 or a target track 314, the logical copy module 160 may issuethe withdraw commands transparently to the user. In other embodiments,the withdraw command may be initiated by the user. For example, the usermay decide that she is done working with the target data set 320, anddelete the target data set 320. Deleting the target data set 320 maycause the logical copy module 160 to issue withdraw commands for theremaining logical copy relationships 316.

In certain embodiments, the logical copy operation 318 may include abackground copy command. Where the logical copy operation 318 includes abackground copy command, the logical copy operation 318 generally firstcauses the creation of logical copy relationships 316 as describedabove. In certain embodiments, the logical copy operation 318 may thencause the data in the source tracks 312 to be physically copied to thetarget tracks 314 as part of a background process. The backgroundprocess may be given a low priority such that the background processdoes not interrupt more important processes on the local site 102. Oncethe background process completes, there will be a separate physical copyof the source data set 310 at the target data set 320. As the backgroundprocess is executed, affected logical copy relationships 316 areremoved. For example, when the data in source track 312 a is copied totarget track 314 a, the logical copy relationship 316 a is removed. Whenthe data in the source tracks 312 a-d is completely copied to targettracks 314 a-d, the logical copy relationships 316 a-d will have allbeen removed.

FIG. 4 shows one embodiment of a logical copy synchronization apparatus400 that includes a receipt module 410, a transfer module 420, and abackup module 430. The logical copy synchronization apparatus 400 mayinclude more or fewer modules in varying embodiments.

In one embodiment, the receipt module 410 may be configured to receive awithdraw command for a logical copy relationship 316 between a localsource track 512 and a local target track 514 (shown in FIG. 5). A localsource track 512 is a source track 312 on the local site 102 that isbeing asynchronously replicated. A local target track 514 is a targettrack 314 on the local site 102 that is being asynchronously replicated.The withdraw command, as explained above, causes the logical copyrelationship 316 between the local source track 512 and the local targettrack 514 to terminate on the local site 102. In certain embodiments,the withdraw command causes a pointer associating the local source track512 and the local target track 514 to be removed.

The logical copy synchronization apparatus 400 may also include atransfer module 420 that sends the withdraw command to a remote sitebuffer 140. The withdraw command may be transferred from the remote sitebuffer 140 on the local site 102 to the remote site 104 that is a mirrorof the local site 102.

The backup module 430 may be configured to set the change indicator 204for one or both of the local source track 512 and the local target track514 in response to receiving the withdraw command for the logical copyrelationship 316 between the local source track 512 and the local targettrack 514. In certain embodiments, when the withdraw command is issuedfor a logical copy relationship 316, the backup module 430 sets thechange indicator 204 for the local target track 514 rather than thelocal source track 512.

FIG. 5 shows an example of how a logical copy synchronization apparatus400 may operate. FIG. 5 shows the logical copy synchronization apparatus400 receiving a withdraw command initiated by a user. As explainedabove, the withdraw command may be initiated by a user when, forexample, the user deletes the target data set 320 that is a copy of thesource data set 310. The logical copy synchronization apparatus 400 maysend the withdraw command to the remote site buffer 140. The mirrormodule 110 will coordinate transferring the withdraw command to theremote site 104, where the withdraw command will be executed,terminating the logical copy relationship 316 between the remote sourcetrack and the remote target track on the remote site 104 that correspondto the local source track 512 and the local target track 514, thusensuring that the remote site 104 is a mirror of the local site 102.

The logical copy synchronization apparatus 400 may also cause the changeindicator 204 to be set for one of the local source track 512 and thelocal target track 514 on the local site 102. As shown in FIG. 5, thelogical copy synchronization apparatus 400 may cause the changeindicator 204 b for the local target track 514 to be set. As explainedabove, the change indicator 204 b may be a bit in a bitmap that isassociated with the local target track 514. In the example of FIG. 5,the bit is flipped from a 0 to a 1, while the change indicator 204 a bitis left at a 0. The withdraw command causes the logical copyrelationship 316 to be withdrawn.

While the above example, and the application, may speak of a simple casewhere there is a withdraw command that affects one logical copyrelationship 316, in many cases the withdraw command will affectmultiple logical copy relationships 316. For example, the withdrawcommand may specify a plurality of local tracks 202 in logical copyrelationships 316 on the local site 102. In certain embodiments, thewithdraw command specifies a range of local tracks 202 for which thelogical copy relationships 316 are to be withdrawn.

The logical copy relationships 316 may be created by an establishcommand. The establish command may be initiated by a user; for example,the user may initiate the establish command by directing the logicalcopy module 160 to create a copy of a particular source data set 310. Incertain embodiments, the establish command that creates the logical copyrelationships 316 may include a background copy command. In certainembodiments, where the establish command included a background copycommand, the logical copy synchronization apparatus 400 may refrain fromsetting change indicators (such as change indicator 204 b or the changeindicator 204 a for the local source track 512 and the local targettrack 514) if the establish command included a background copy command.

FIGS. 6A, 6B, and 6C show how the method described above may help ensureconsistency between the local site 102 and the remote site 104 in theevent of a suspension causing the remote site buffer 140 to be lost.FIG. 6A shows a remote site buffer 140 with local tracks 202 a-c to besent to the remote site 104, along with a withdraw command. As explainedabove, the local tracks 202 a-c may have been changed on the local site102. Data representing these changes is placed in the remote site buffer140 to ensure that the remote site 104 executes appropriate commandsthat make changes to remote tracks corresponding to local tracks 202 a-con the remote site 104.

A withdraw command is also in the remote site buffer 140. In thisexample, the withdraw command is directed to logical copy relationship316 a between local source track 312 a and local target track 314 a onthe local site 102. As explained above, the withdraw command willtypically be for multiple logical copy relationships 316, but only asingle logical copy relationship is discussed here for ease ofunderstanding. The single case discussed here is easily extended to thecase of multiple logical copy relationships 316.

When the withdraw command is received at the local site 102, the logicalcopy relationship 316 a on the local site 102 is terminated (as seen bythe “X” in FIG. 6B). In addition, the change indicator 204 b may be setfor the local target track 314 a. In certain embodiments, the changeindicator 204 a for the local source track 312 a is changed instead. Inother embodiments, both change indicators 204 a and 204 b are set.

The local source track 312 a on the local site 102 may correspond to aremote source track 612 a on the remote site 104, and the local targettrack 314 a may similarly correspond to a remote target track 614 a.When the logical copy relationship 316 a was established on the localsite 102, the remote logical copy relationship 616 a was similarlyestablished in response on the remote site 104. Following execution ofthe withdrawal command on the local site 102, causing the logical copyrelationship 316 a to be terminated, the remote logical copyrelationship 616 a remains on the remote site 104. Thus, the data on thelocal site 102 and the remote site 104 may be out of synchronizationuntil the withdraw command is transferred from the remote site buffer140 to the remote site 104.

In the event of a suspension, the data in the remote site buffer 140 maybe lost. The changes that were in the remote site buffer 140 at thattime were therefore not transmitted to the remote site 104 is not a truemirror of the local site 102 as a result. The mirror module 110 mayrebuild the remote site buffer 140, thereby creating the rebuilt remotesite buffer 640 shown in FIG. 6A.

In certain embodiments, the withdraw command itself may be permanentlylost when the suspension occurs. In response to the suspension, themirror module 110 may build the rebuilt remote site buffer 640. Usingthe change indicators 204, the mirror module 110 may determine that thelocal tracks 202 a-c were changed on the local site 102 and cause thedata contained therein to be inserted in the rebuilt remote site buffer640. In addition, the mirror module 110 may also see the changeindicator 204 b for the local target track 314 a has been set and causedthe data contained therein to be inserted into the rebuilt remote sitebuffer 640. As can be seen in the example shown in FIG. 6A, the withdrawcommand itself may be permanently lost and may not be placed in therebuilt remote site buffer 640.

The mirror module 110 may then cause the data in the rebuilt remote sitebuffer 640 to be written on the remote site 104. As a result, datacorresponding to the local target track 314 a may be written to theremote target track 614 a. This write operation causes the remotelogical copy relationship 616 a to be terminated, which synchronizes theremote source track 612 a, the remote target track 614 a, and the remotelogical copy relationship 616 a with the local source track 312 a, localtarget track 314 a, and the logical copy relationship 316 a.

Thus, the mirror module 110 may detect a suspension of the mirrorrelationship between the local site 102 and the remote site 104 thatcauses the withdraw command in the remote site buffer 140 to bediscarded. The withdraw command may be discarded because it was storedin volatile memory which was discarded due to the suspension. The mirrormodule 110 may also determine that data in local tracks 202 that havetheir associated change indicators 204 set may be inconsistent at theremote site 104.

The mirror module 110 may send the data in the local tracks 202 havingthe change indicator 204 set to be written at corresponding remotetracks on the remote site 104. The write operation causes any remotelogical copy relationship 616 a between a remote source track 612 a anda remote target track 614 a to be withdrawn at the remote site 104. Asexplained above, this may ensure that a logical copy relationship 316 ais kept in synch with a remote logical copy relationship 616 a even whenthe withdraw command is lost and never sent to the remote site 104.

In certain embodiments, the change indicators 204 are bitmap entriesthat may be used to track when a change has been made to a local track202; that is, the bit entry is flipped if there is a change to thecontent of the associated local track 202. The bit entry may thusindicate that a change has occurred, that the associated local track 202may be out of synchronization with the remote site 104, and that thislocal track 202 should be resynchronized in the event of a suspension.In certain embodiments, the present invention may flip the bit entryeven though there is no change to the local track 202 if the local track202 is part of a logical copy relationship that has been withdrawn onthe local site 102.

FIG. 7 shows one embodiment of a method 700 for maintainingsynchronization of logical copy relationships 316 on a local site 102and a remote site 104. The method may begin with receiving 702 awithdraw command for a logical copy relationship 316 on the local site102. The withdraw command may cause logical copy relationships 316between local source tracks 512 and local target tracks 514 to terminateon the local site 102. The method may further comprise executing 704 thewithdraw command for the logical copy relationships 316 on the localsite 102 and sending 706 the withdraw command to a remote site buffer140 on the local site 102. In such embodiments, the withdraw command,absent a suspension, may be transferred from the remote site buffer 140to the remote site 104 for execution.

The method 700 may involve determining 712 whether the establish commandwhich created the logical copy relationships 316 being withdrawnincluded a background copy command. If the establish command did includea background copy command, the method may continue without takingfurther steps to synchronize the logical copy relationships 316 on thelocal site 102 with those on the remote site 104. In certainembodiments, if the establish command included a background copycommand, the logical copy synchronization apparatus 400 does not performthe step 708.

In certain embodiments, if the establish command did not include abackground copy command, the method 700 may also include setting 708 achange indicator 204 for the local source tracks 512 and/or the localtarget tracks 514 in the logical copy relationships 316 that are beingwithdrawn by the withdraw command. In certain embodiments, only thechange indicator 204 for the local target tracks 514 are set. Settingthe change indicator 204 indicates that the associated local track 202has changed on the local site 102. The change indicator 204 may be setwhen the withdraw command is received even though the data in the localtarget track 514 has not actually changed. Thus, in certain embodiments,a change indicator 204 is set, indicating that the associated localtrack 202 has changed, in response to receiving the withdraw command forthe logical copy relationship regardless of whether or not the data inthe local track 202 has actually changed.

The method 700 may also involve monitoring 710 the system for asuspension which causes the remote site buffer 140 to be discarded. Ifthere is no suspension, the steps of receiving withdraw commands,placing those withdraw commands in the remote site buffer 140, andsetting change indicators 204 may continue.

In certain embodiments, the method 700 may continue with determiningthat data in the local tracks 202 having the change indicators 204 setmay be inconsistent at the remote site 104. In such situations, onlysome of the data in local tracks 202 may be inconsistent at the remotesite 104. The method 700 may involve sending data in the local tracks202 having the change indicator 204 set to be written at the associatedremote tracks on the remote site 104. As a result, those remote tracksthat are in remote logical copy relationships 616 a which wereterminated by the withdraw command on the local site 102 will bewithdrawn from the remote logical copy relationships 616 on the remotesite 104 when the data is written to a remote track (whether the remotesource track 612 or the remote target track 614) in the remote logicalcopy relationship 616.

The embodiments may be practiced in other specific forms. The describedembodiments are to be considered in all respects only as illustrativeand not restrictive. The scope of the invention is, therefore, indicatedby the appended claims rather than by the foregoing description. Allchanges which come within the meaning and range of equivalency of theclaims are to be embraced within their scope.

What is claimed is:
 1. A computer program product stored on anon-transitory computer readable medium, the computer program productconfigured to: maintain a bitmap for storage devices on a local site,each storage device comprising a plurality of local tracks, wherein eachlocal track is associated with a change indicator in the bitmap thatindicates a change in data when set; receive a withdraw commandinitiated by a user for a logical copy relationship between a localsource track and a local target track on a local site, wherein thewithdraw command causes the logical copy relationship between the localsource track and the local target track to terminate on the local site;send the withdraw command to a remote site buffer; transfer the withdrawcommand from the remote site buffer to the remote site; and for one orboth of the local source track and the local target track in the logicalcopy relationship that is a subject of the withdraw command, set thechange indicator in response to receiving the withdraw command for thelogical copy relationship.
 2. The computer program product of claim 1,further configured to: detect a suspension of a mirror relationshipbetween the local site and the remote site, the suspension causing thewithdraw command in the remote site buffer to be discarded.
 3. Thecomputer program product of claim 2, further configured to: determinethat data in local tracks having the change indicator set may beinconsistent at the remote site, wherein the local tracks comprise localsource tracks and local target tracks.
 4. The computer program productof claim 3, further configured to: send data in the local tracks havingthe change indicator set to be written at remote tracks on the remotesite, such that a remote logical copy relationship between a remotesource track and a remote target track is withdrawn at the remote site.5. The computer program product of claim 1, wherein the withdraw commandis for a plurality of local source tracks and local target tracks inlogical copy relationships on the local site.
 6. The computer programproduct of claim 1, further configured to: determine if an establishcommand creating the logical copy relationship comprised a backgroundcopy command.
 7. The computer program product of claim 6, furtherconfigured to: refrain from setting the change indicator in response toreceiving the withdrawal command where the establish command comprised abackground copy command.
 8. A method comprising the steps of:maintaining, by use of a processor, a bitmap for storage devices on alocal site, each storage device comprising a plurality of local tracks,wherein each local track is associated with a change indicator in thebitmap that indicates a change in data when set; receiving a withdrawcommand initiated by a user for a logical copy relationship between alocal source track and a local target track on a local site, wherein thewithdraw command causes the logical copy relationship between the localsource track and the local target track to terminate on the local site;sending the withdraw command to a remote site buffer; transferring thewithdraw command from the remote site buffer to the remote site; and forone or both of the local source track and the local target track in thelogical copy relationship that is a subject of the withdraw command,setting the change indicator in response to receiving the withdrawcommand for the logical copy relationship.
 9. The method of claim 8,further comprising: detecting a suspension of a mirror relationshipbetween the local site and the remote site, the suspension causing thewithdraw command in the remote site buffer to be discarded.
 10. Themethod of claim 9, further comprising: determining that data in localtracks having the change indicator set may be inconsistent at the remotesite.
 11. The method of claim 10, further comprising: sending data inthe local tracks having the change indicator set to be written at remotetracks on the remote site, such that a remote logical copy relationshipbetween a remote source track and a remote target track is withdrawn atthe remote site.
 12. The method of claim 8, wherein the withdraw commandis for a plurality of local tracks in logical copy relationships on thelocal site.
 13. The method of claim 8, further comprising: determiningwhether an establish command creating the logical copy relationshipcomprised a background copy command.
 14. The method of claim 13, furthercomprising: refraining from setting the change indicator in response toreceiving the withdrawal command where the establish command comprised abackground copy command.
 15. A system comprising: a mirror moduleconfigured to create a mirror relationship between a local site and aremote site communicatively connected to the local site, the mirrormodule also configured to store change data representing changes tolocal data in a remote site buffer on the local site; a bitmap forstorage devices on the local site, each storage device comprising aplurality of local tracks, wherein each local track is associated with achange indicator in the bitmap that indicates a change in data when set;a receipt module configured to receive a withdraw command initiated by auser for a logical copy relationship between a local source track and alocal target track on the storage devices on the local site, wherein thewithdraw command causes the logical copy relationship between the localsource track and the local target track to terminate on the local site;the mirror module further configured to send the withdraw command to theremote site buffer and transfer the withdraw command from the remotesite buffer to the remote site; and a backup module configured to setthe change indicators in the bitmap for one or both of the local sourcetrack and the local target track in the logical copy relationship thatis a subject of the withdraw command in response to receiving thewithdraw command for the logical copy relationship.
 16. The system ofclaim 15, the mirror module further configured to detect a suspension ofthe mirror relationship, the suspension causing the withdraw command inthe remote site buffer to be discarded.
 17. The system of claim 16, themirror module further configured to: retrieve data from the local targettracks with the change indicators set in the bitmap; and send the datain the local target tracks to be written at the remote site.
 18. Anapparatus comprising: a bitmap for storage devices on the local site,each storage device comprising a plurality of local tracks, wherein eachlocal track is associated with a change indicator in the bitmap thatindicates a change in data when set; a receipt module that receives awithdraw command initiated by a user for a logical copy relationshipbetween a local source track and a local target track on a local site,wherein the withdraw command causes the logical copy relationship toterminate on the local site; a transfer module that sends the withdrawcommand to a remote site buffer for a remote site that is a mirror ofthe local site; a mirror module that sends the withdraw command to theremote site buffer and transfers the withdraw command from the remotesite buffer to the remote site; and a backup module that sets, for oneor both of the local source track and the local target track in thelogical copy relationship that are a subject of the withdraw command, achange indicator in response to receiving the withdraw command for thelogical copy relationship.
 19. The apparatus of claim 18, wherein thebackup module determines if the establish command creating the pluralityof logical copy relationships comprised a background copy command, andrefrains from setting the change indicator for the local target tracksin response to determining that the establish command comprised abackground copy command.